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La exposicion de los estudiantes a la educacion basada en competencias 
Resumen: Recientemente, la mayorfa de los estados en los Estados Unidos han revisado la 
poliza educativa con metodos que han proveldo flexibilidad para alejarse del sistema de 
unidades de Carnegie, que se centra en los creditos y el "tiempo en el asiento o escritorio", 
hacia pollticas basadas en competencias que conectan el progreso del estudiante con la materia 
del contenido. Sin embargo, hay poca informacion recopilada sistematicamente sobre como se 
implementa la educacion basada en competencias, lo que dificulta evaluar el impacto en los 
resultados de los estudiantes. Utilizando datos de 600 estudiantes de los grados 9-12 con 
tecnicas anallticas con factores confirmatorios, informamos los resultados iniciales de 
confiabilidad y validez de la administration initial de una encuesta disenada para capturar la 
exposicion de los estudiantes a elementos que se han descritos como esenciales para un 
estudiante basado en competencias centradas para el aprendizaje y la instruction. Estos 
elementos incluyen la progresion basada en el la maestria, la personalization, la evaluation 
flexible y el desarrollo de habilidades y disposiciones especificas. Los resultados sugieren que la 
encuesta ofrece una manera de medir y estudiar con fiabilidad la variation en la 
implementation de la educacion basada en competencias. Es importante destacar que la 
encuesta proporciona una forma de captar la implementation desde la perspectiva del 
estudiante, aprovechando el hecho de que los informes de los estudiantes sobre sus 
experiencias en el salon pueden ser una fuente particularmente confiable de informacion sobre 
la practica de instruction. 

Palabras-clave: educacion basada en competencias; educacion secundaria; modelos de 
ecuaciones estructurales; encuestas 

A exposi§ao os estudantes a educacao baseada em competencias 
Resumo: Recentemente, a maioria dos estados dos Estados Unidos ter revisto a politica 
educacional com metodos que proporcionaram flexibilidade para se afastar das unidades 
do sistema Carnegie, que incide sobre o credito e "tempo no banco ou mesa" para pollticas 
com base em competencias que ligam o progresso do estudante com teor de materia. No 
entanto, pouca informaqao e sistematicamente coletadas sobre como a educaqao baseada 
em competencias e implementado, o que torna diflcil avaliar o impacto sobre os resultados 
dos alunos. Usando dados de 600 alunos 9.12 com tecnicas anallticas com fator de 
confirmaqao, relatamos os resultados initials de confiabilidade e validade da administraqao 
initial de uma pesquisa projetado para capturar a exposiqao aluno elementos que foram 
descritos como essential para uma aprendizagem baseada em competencias centrado no 
aluno e instruqao. Esses elementos incluem progressao com base em perlcia, 
personalizaqao, avaliaqao flexlvel e desenvolvimento de habilidades e disposiqSes 
especificas. Os resultados sugerem que a pesquisa fornece uma maneira de medir e estudar 
a variaqao na implementaqao da educaqao baseada em competencias de forma confiavel. E 
importante ressaltar que a pesquisa fornece uma maneira de capturar a implementaqao da 
perspectiva do estudante, aproveitando o facto dos relatorios de estudantes sobre as suas 
experiencias em sala de aula pode ser uma fonte particularmente confiavel de informaqoes 
sobre a pratica de ensino. 

Palavras-chave: educacao baseada em competencias; educacao secundaria; modelagem equacao 
estrutural; pesquisas 
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Introduction 

Many states are beginning to move away from policies under which student advancement is 
based on the traditional Carnegie unit system, with its focus on credits and “seat time,” and toward 
policies that provide schools with the flexibility to link advancement to a student’s mastery of 
content (Marion & Leather, 2015; Scheopner-Torres, Brett, & Cox, 2015). The movement toward 
competency-based approaches to learning and instruction addresses a recommendation from the 
National Center on Time & Learning (2011) that learning programs should factor in the different 
amounts of time students need to achieve proficiency and should consider basing student 
advancement on proficiency, rather than on the amount of time students spend sitting in a 
classroom. This shift also reflects the position that students are more engaged in their learning when 
they are granted greater flexibility in how they earn credit (United States Department of Education, 

2011) . Stakeholders representing a variety of perspectives are increasingly embracing the notion that 
student-centered (Jobs For the Future, 2014) reforms like competency-based education will yield 
increased achievement and better preparation for college and the workforce. Similar claims have 
been made about other so-called student-centered approaches, including both personalized learning 
(Bill & Melinda Gates Foundation, 2015) and deeper learning (Alliance for Excellent Education, 

2012 ) . 

Both educators and researchers report that implementation of competency-based education 
(also referred to as proficiency-based education, or simply competency education) varies widely 
(Scheopner-Torres et al., 2015; Steele et al., 2014). Implementation challenges include carrying out 
necessary yet substantial shifts in instruction, assessment, and grading in the classroom; addressing 
student misconceptions about such shifts in school practices and policies; and supporting student 
development of the skills and dispositions necessary for student success under a competency-based 
model (Le, Wolfe, & Steinberg, 2014; Scheopner-Torres et al., 2015). Further, there is a dearth of 
systematically collected information about how and to what extent instructional practices reflecting 
elements described as essential to competency-based education are being implemented, making it 
difficult to investigate how competency-based reforms are or are not leading to changes in student 
outcomes (Freeland, 2014; Haynes et al., 2016; Le et al., 2014). Moreover, and perhaps surprisingly 
given the emphasis on student voice in the literature on competency-based education (Le et al., 
2014), relatively little is known about how students perceive and experience changes to school 
policies and instmctional practices. 

In what follows, we report results from the pilot administration of a survey designed to 
capture student exposure to elements currently described as essential to competency-based 
education. These elements, which are multi-dimensional, include mastery-based progression, 
personalization, flexible assessment, and the development of specific skills and dispositions (Sturgis, 
2016). Valid and reliable measures of student exposure to elements of competency-based education 
are needed if researchers are to more carefully investigate the relationship between implementation 
and student outcomes. Given that student reports about their classroom experiences appear to be a 
reliable source of information about instructional practice (Kane & Staiger, 2012), measures based 
on student-reported exposure to competency-based education hold promise for facilitating research 
of this nature. The results indicate that this survey reliably measures various dimensions underlying 
each of the hypothesized elements of competency-based education, providing support for the future 
use of the survey by both researchers and practitioners to examine implementation and its 
association with student outcomes. Yet, the results also raise questions about whether competency- 
based education is a distinct, bounded model for secondary education comprising all four elements 
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described as essential to the approach. We conclude by discussing the implications of these findings 
for practice, for research, and for policy. 

Renewed Interest in Competency-Based Education 

In recent years, competency-based education has garnered renewed attention as a reform to 
increase graduation rates and to ensure that students are prepared for postsecondary success 
(Scheopner-Torres et al., 2015). The theory is that a larger share of students will ultimately reach 
proficiency in a given content area if they are given the freedom to advance at their own pace and if 
their learning experiences are tailored to their needs and interests (Lewis et al., 2014; Sturgis & 
Patrick, 2010). 

Several generations of competency-based education have been documented in the research 
literature, tracing back to at least the 1960s (Brown, 1994). Current interest in this reform continues 
a long tradition of challenging the idea that learning happens at a particular pace and place for all 
students (Brown, 1994; Dewey, 1938; Le et al., 2014; Spady & Mitchell, 1977). Until recently, 
however, competency-based education was primarily considered a niche approach targeting 
vocational education and the adult learning segment of the higher education system (Ford, 2014). 
Recent advances in online learning, learning analytics, and adaptive learning technology, combined 
with growing demand for demonstrable college and career readiness outcomes among high school 
graduates, have prompted renewed and expanded interest in the development of major competency- 
based initiatives (Ford, 2014). Marking a shift from earlier iterations of competency-based education, 
policymakers have been more responsive to the present movement (Steele et al., 2014). As of 2014, 
42 states had authorized at least some flexibility for local education agencies to link the completion 
of academic credit to demonstration of proficiency rather than only to seat time (Carnegie 
Foundation for the Advancement of Teaching, 2014). 

Over time, competency-based education has been described using a variety of frameworks, 
all of which share some degree of overlap. Several decades ago, Spady (1977) described competency- 
based education as consisting of six critical elements including outcomes, time, instmction, 
measurement, certification, and program adaptability. More recently, based on interviews with 
educators and a literature review of recent developments in the competency-based education 
movement, Steele and colleagues (2014) articulated three core principles, including flexible pacing, 
opportunities to personalize learning, and the requirement that students demonstrate proficiency 
(and earn credit) by applying knowledge and skills. Most recently, Sturgis (2016) outlined the 
following four elements as essential to the approach: student advancement through demonstration 
of mastery based on explicit and measurable learning objectives reflecting important competencies; 
personalized instmction that provides a student with customized supports and opportunities to 
engage in anytime/anywhere learning; multiple modes of, and multiple opportunities for, assessment 
that allow a student to apply his or her learning; and an emphasis on the development of specific 
skills and dispositions critical for success in a learning environment where students are expected to 
play a more prominent role in directing their own learning. The research described below, and the 
student survey used in the research, focus on the four essential elements described by Sturgis (2016). 
These elements share substantial overlap with the components of competency-based education 
articulated both in other recent literature (for example, see Competency Works, 2014; Patrick & 
Sturgis, 2013; Scheopner-Torres et al., 2015; Steele et al., 2014) and in legislation and policy in states 
that have adopted competency-based education. 

Alongside renewed interest in the promise of competency-based education has been a “wild 
proliferation” (Weise, 2014) of interest in student-centered reforms broadly, perhaps especially in 
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the realm of education philanthropy. For example, the past several years have also witnessed 
growing enthusiasm for other student-centered models, including personalized learning (Bill & 
Melinda Gates Foundation, 2015) and deeper learning (Alliance for Excellent Education, 2012; 
William & Flora Elewlett Foundation, 2014), as a path to better student college and workforce 
preparation. While the terms used to name these overlapping approaches vary, they share in 
common some or all of the elements used to describe competency-based education. This suggests 
growing consensus about the core of student-centered educational reform broadly, and the centrality 
of these elements to student-centered reforms. Nonetheless, the variation in terminology is 
noteworthy, particularly to the extent that it both reflects and contributes to imprecision, if not 
confusion, in how the elements of competency-based education and other student-centered 
approaches are defined—an issue that is not new, but was recognized some three decades ago by 
Spady and colleagues (Spady, 1977; Spady & Mitchell, 1977). Echoing this lack of precision, both in 
language and meaning, is wide variability within and across schools and districts in how competency- 
based policies and practices are conceived and implemented. Ultimately, these issues pose significant 
challenges for efforts to understand the relationship between the implementation of competency- 
based education and student outcomes. 

Implementation of Competency-Based Education and Student Outcomes 

While many are enthusiastic about the promise of competency-based education, the bridge 
between a promising idea and impact on student outcomes is implementation—and innovations are 
seldom implemented as intended (Berman & McLaughlin, 1976). Although there is general 
agreement among proponents of competency-based education about its essential elements, 
implementation varies substantially across states and districts (Steele et al., 2014; Scheopner-Torres 
et al., 2015; Stump & Silvernail, 2015). Competency-based policies range from those that simply 
allow flexibility in awarding credit, to policies that completely transform the education system “from 
a time-based system to a learning-based system” (Sturgis & Patrick, 2010, p. 4). Some states require 
implementation of a competency-based diploma system in all districts; in other states districts can 
choose to implement elements of competency-based education, but state policies neither require nor 
restrict implementation of the reform. 

States also vary in how they implement competency-based grading policies and graduation 
requirements (Sturgis, 2014). In some contexts, districts are required to make competency-based 
credits available, but in other places districts are allowed to decide whether these will be offered. 
Some states specify how students will demonstrate competency (e.g., scores on statewide or national 
tests, portfolios), while others leave it to the discretion of the district to determine how 
competencies are measured. Certain states allow credits to be awarded for demonstration of 
competency in any course, but others only offer this in certain subjects. 

Beyond variation in the nature of state competency-based policies, how these policies are 
implemented at the local level also differs. For instance, one study (Steele et al., 2014) of 
implementation in several districts throughout the United States documented variability in which 
competency-based education elements districts chose to implement. Some districts focused primarily 
on the assessment of proficiency on competencies, but others focused on personalization of 
learning, including student choice and flexible pacing. This study also found variability across sites in 
a number of other areas including: the extent to which learning experiences that occurred outside of 
school hours and off of school grounds were counted toward the fulfillment of course requirements, 
the criteria used to determine student proficiency on a specific standard or competency, and the use 
of technology to facilitate online learning opportunities and operate learning management systems. 
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In order to develop a body of empirical evidence about how competency-based education is 
related to student outcomes, it will be necessary to continue building a better understanding of 
variability in implementation, including the development of valid and reliable ways to measure 
implementation fidelity. In particular, practitioners and policymakers would benefit from a better 
understanding of how students’ experiences with competency-based education are consistent with 
current descriptions of the model and its component elements. As research about the use of student 
perception surveys to gather information about students’ classroom experiences has demonstrated, 
student surveys may provide more consistent results than classroom observations about what is 
happening in the classroom (Balch, 2012; Ferguson, 2012; Kane & Staiger, 2012), offering an 
important source of information about implementation. 

Researchers describe five aspects of implementation fidelity (Dane & Schneider, 1998; 
Durlak & DuPre, 2008; Dusenbury, Brannigan, Falco, & Hansen, 2003). Adherence references 
whether specific features of a program are implemented as prescribed by the program model. 
Exposure reflects the amount of the program delivered and/or the extent to which those served by 
the program were exposed to each element of the model. Quality refers to how well the program is 
implemented, including the caliber of the delivered model features. Program differentiation indicates the 
degree to which the essential elements of a program are distinguishable from each other and from 
other programs. Finally, responsiveness reflects the manner in which those exposed react to or engage 
in the program. 

At the student level, the survey used in this research captures information about student 
exposure to each of the hypothesized elements of competency-based education; this includes 
information about whether the student is exposed to instructional practices associated with each 
element and information about the frequency with which the student is exposed to such practices 
(the survey does not, however, capture duration, or the amount of time a student is exposed to these 
practices). At the school level, results can provide information about adherence, or the extent to 
which specific elements prescribed in the current literature about competency-based education are 
implemented. Results from this survey can also be used to explore program differentiation, or the 
extent to which hypothesized essential elements of competency-based education are distinguishable 
from each other and whether, as a whole, these elements constitute a bounded, coherent model that 
is distinct from other programs—an issue of particular emphasis in this research. 

Conceptual Framework 

This research offers one of the first empirically based attempts to operationalize and 
measure what have been proposed as essential elements of competency-based education. 
Information about students’ exposure to competency-based education can support practitioners and 
policymakers in making more informed decisions about implementation, and can facilitate research 
addressing the relationship between implementation and student outcomes. 

The limited empirical research base on the implementation and outcomes of competency- 
based education, along with the absence of conceptual clarity about what key elements of 
competency-based education should look like in practice, pose challenges for measurement. Our 
approach in the current study is informed by the position that validating new constmcts involves an 
iterative, ongoing process of refining both theory and measurement (Westen & Rosenthal, 2003). 

We begin this process by drawing from the existing literature on competency-based education to 
inform the constmcts we study in the current research—constmcts intended to capture the 
hypothesized key elements of competency-based education as measured by their underlying 
dimensions. 
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At the core of competency-based education is an approach to teaching and learning that 
allows students to advance in school based on demonstrated mastery of important content, typically 
as defined through measurable learning objectives reflecting competencies required for success in 
college, career, and life (Le et al., 2014). One dimension of progression through demonstration of 
mastery is competency-based pacing (CB PACE; see Figure 1), which involves student exposure to 
instructional practices that allow them to work at their own level and pace, advancing to the next 
level of study based on demonstrated mastery of skills rather than on their age or number of hours 
they’ve spent in class (Le et al., 2014). A second dimension is competency-specific feedback (CB FEED), 
which is intended to ensure that students understand what is expected of them, including how their 
efforts will be assessed and what mastery (or proficiency—the terms are often used interchangeably) 
looks like. See Appendix Table Al for a description of each dimension and the indicators of each. 

One of the more elusive elements of competency-based education—and other student- 
centered reforms—is student personalization. The notion of personalization in K-12 education 
broadly is both increasingly popular and variably defined (for example, see Le et al., 2014). In the 
competency-based education literature, dimensions of personalization typically include personalised 
options for practicing and applying skills (PERS APP; this is sometimes referred to as “anytime, 
anywhere” learning), along with exposure to personalised instructionalpractices (PERS INSTR) that 
respond to the needs of individual students. 

With respect to student assessment, the literature on competency-based education also 
emphasizes flexible assessment (FLEX). Flexible assessment reflects the idea that students should have 
multiple and varied opportunities to demonstrate mastery of important skills as part of a summative 
assessment system (i.e., not simply quizzes or exams). An emphasis on flexible assessment also 
reflects a belief that students may require multiple attempts to demonstrate mastery, such that falling 
short of mastery on the first attempt is interpreted not as a failure but as indicating a need for 
additional practice and support before subsequent attempts. 

Finally, current literature suggests the need to support student development of specific skills 
and dispositions hypothesized as necessary for student success within a competency-based 
framework for learning and instruction. Although we use the phrase skills and dispositions here, we 
note that the literature uses a range of overlapping terms, including, for example, work-study habits, 
non-cognitive skills, and socio-emotional skills. Amidst an era of great interest in students’ social and 
emotional well-being, a range of skills and dispositions have been proposed in the literature on 
competency-based education as important for student success under this model. We focus on five 
dimensions reflecting the development of student skills and dispositions. The first is exposure to 
instructionalpractices that encourage respect for others (RSPCT; Kallick & Costa, 2009). The second is 
exposure to instructionalpractices that encourage students to persevere (PRSVR; Farrington et al., 2012) even 
when faced with a challenge. We also focus on exposure to instructionalpractices that encourage students to 
take responsibility for their learning (RSPNS; Lewis et al., 2014), as well as student demonstration of ownership 
over their learning (OWN; Le et al., 2014). Finally, we focus on student exposure to instructionalpractices 
that encourage peer collaboration (COLL; Kallick & Costa, 2009). 

In this research, we model latent factors representing student exposure to the dimensions 
underlying hypothesized elements of competency-based education. Using data collected via a survey 
designed to capture student feedback about the implementation of competency-based education in 
their courses, we address three research questions: (1) Can dimensions underlying hypothesized 
elements of competency-based education be reliably measured?; (2) To the extent that dimensions of 
competency-based education elements can be reliably measured, how are they associated with one 
another?; and (3) To what extent do the results provide evidence of construct validity? 
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We investigate these questions using confirmatory factor analysis (CFA), which can show 
whether the observed variables hypothesized to measure an unobserved construct demonstrate a 
level of shared variance high enough to suggest that those items represent a common underlying 
factor (i.e., the constmct of interest). Per the conceptual framework depicted in Figure 1, we 
consider three potential ways of representing student exposure to dimensions underlying the 
hypothesized elements of competency-based education. We begin with a correlated traits model 
(Model A) in which various dimensions of instructional practice underlying the four key elements of 
competency-based education are represented by the 10 latent factors depicted in Figure 1. In the 
correlated traits model, the 10 latent factors are considered to be related to varying degrees but no 
attempt is made to measure one or more higher-order factors. 




Model C 



Figure 1. Conceptual framework for the study. 

Note: The conceptual framework hypothesizes three potential manifestations of competency-based education 
implementation, as represented by three latent factor models. In Model A, dimensions underlying hypothesized elements 
of competency-based education are modeled as a set of interrelated latent factors, although no underlying explanatory 
structure is imposed on the interrelationships In Model B, it is hypothesized that interrelationships among these 
dimensions are explained by a set of second-order factors representing the hypothesized elements of competency-based 
education. Model C imposes a stmcture by which interrelationships among the 10 individual dimensions are explained 
by an overall competency-based education second-order factor. In the figure, shaded ellipses depict first-order latent 
factors, and unshaded ellipses depict second-order latent factors. Each first-order latent factor is measured via observed 
indicators (see Table 1 and Appendix Table Al), which are not depicted. 
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To explore whether dimensions underlying the hypothesized essential elements of 
competency-based education can be represented in a manner consistent with the current description 
of competency-based education as a distinct program model, we also consider 2 second-order factor 
models. In Model B (see Figure 1), which is most consistent with current descriptions, the 
dimensions were grouped as indicators of second-order factors representing student exposure to the 
hypothesized key elements of competency-based education. This model assumes that the second- 
order factors (elements of competency-based education) explain the correlations among the first- 
order factors (dimensions of each element). We also explore an alternative model (Model C; see 
Figure 1) that assumes the correlations among the 10 first-order factors are explained by one overall 
competency-based education second-order factor rather than by a set of component elements. 

Method 

Participants 

After securing Institutional Review Board approval, the data for this study were collected in 
the spring of 2016 at two high schools in the Northeast, including a small rural high school and a 
midsize suburban high school. Approximately 80% of students in grades 9-12 (#=599) at the two 
schools completed the survey; missing responses primarily reflected student absence on the date of 
survey administration (#=83), including a field trip for approximately half of students in grade 12 at 
one high school. Students in grade 9 made up 33% of the sample, and students in grade 10 made up 
24%, followed by students in grades 11 (29%) and 12 (14%). Just over half (52%) of the sample was 
female. A majority of the sample expected to complete a bachelor’s degree (39%) or above (42%). 

Survey Development 

We developed this survey in the context of a research-practice partnership that included 
teachers, school and district administrators, state department of education staff, and researchers. The 
work of the partnership, including the development of the survey, was funded by the Institute of 
Education Sciences through the Regional Educational Laboratory (REL) program and was facilitated 
by researchers at REL Northeast and Islands, one of 10 RELs across the country. The overall goal 
of the REL partnerships is to support practitioners’ and policymakers’ use of empirical evidence to 
inform policy and practice and to improve student outcomes. Partnership research is guided by the 
needs of these stakeholders who, in the case of this survey, identified the need to capture student 
feedback about how competency-based education is actually being implemented. Resources 
developed through the partnership are freely available and no organization or individual profits from 
their use. The survey used in this research, the Competency-Based Learning Survey for Students (Ryan & 
Cox, 2016), can found at https: / /nces.ed.gov/pubsearch/pubsinfo.asp?pubid=REL2016165 . 

We used seminal survey development resources, including Fowler (2008), Rea and Parker 
(2005), and Wright and Marsden (2010), to guide the development of the survey instrument. Survey 
development relied on close collaboration between the researchers and a six-member advisory 
committee made up of practitioners from schools implementing competency-based education along 
with researchers having expertise in competency-based approaches to learning and instruction at the 
secondary level. Ongoing collaboration with the advisory committee was critical to ensure that the 
survey items would elicit useful, valid, and reliable information about student exposure to 
competency-based education policies and practices. Practitioner involvement was especially valuable 
for ensuring that the item language would be meaningful for students across a variety of state 
contexts, particularly given variability in terminology surrounding competency-based education. 

Several steps were taken to establish face (i.e., Does the instmment measure what it is 
intended to measure?) and content (i.e., Does the instrument tap into the various dimensions of the 
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specific constructs of interest?) validity. First, in collaboration with the advisory committee, we 
developed an initial pool of survey items aimed at providing complete coverage of the hypothesized 
constructs. The initial item pool included more items than were necessary to ensure complete 
coverage. The advisors reviewed all draft items, providing feedback on each item in response to a 
number of specific prompts. This process was used to evaluate the items’ face validity and content 
validity. We refined the item pool in response to this feedback, and we cycled through this iterative 
process of review and revision until the item pool covered the constructs completely and constituted 
a survey that could be administered without overburdening the survey respondents or the school 
staff that would administer the survey. The refined item pool formed the first draft of the survey. 

We then administered the initial survey draft to seven students and conducted cognitive 
interviews with these students. Cognitive interviewing is a method for identifying and correcting 
problems with surveys that involves administering a draft survey to a respondent while concurrently 
interviewing him or her to determine if the survey items are eliciting the information that the 
researcher intends (Beatty & Willis, 2007). During cognitive interviews respondents were asked to 
think aloud as they completed the survey, thereby making explicit their interpretations of 
instructions, items, and response choices, as well as their decision points. Information gathered 
during the cognitive interviews was analyzed and discussed with the advisory committee to improve 
the clarity, relevance, length, and coverage of survey items. 

Measures 

We describe below each of the 10 latent factors we modeled to represent different 
dimensions of the hypothesized elements of competency-based education. Descriptive statistics for 
the hypothesized indicators of each latent factor are presented in Table Al. 

Mastery-based progression. Two latent factors reflect dimensions of mastery-based 
progression: competency-based pacing (CB PACE; five items), or exposure to instructional practices 
associated with the opportunity to progress through demonstration of mastery, and competency-specific 
feedback (CB FEED; three items), or exposure to guidance and feedback from teachers about how 
student mastery of important content will be determined. 

Personalization. Dimensions of personalization are reflected through two latent factors. 
The first latent factor reflects exposure to personalised options for practicing and applying skills (PERS 
APP; five items), and the second reflects exposure to personalised instructionalpractices based on the 
needs of individual students (PERS INSTR; six items). 

Flexible assessment. The flexible assessment dimension (FLEX; seven items) reflects 
exposure to multiple and varied opportunities to demonstrate learning. 

Skills and dispositions. Five underlying dimensions reflect the skills and dispositions 
hypothesized as important for student success under competency-based education. These 
dimensions include: exposure to instructionalpractices that encourage respect for others (RSPCT; three 
items); exposure to instructionalpractices that encourage students to persevere even when faced with a 
challenge (PRSVR; three items); exposure to instructionalpractices that encourage students to take 
responsibility for their learning (RSPNS; three items); student demonstration of ownership over learning (OWN; 
three items); and exposure to instructionalpractices that encourage peer collaboration (COLL; three items). 

Analysis 

We used confirmatory factor analysis (CFA) to examine the fit between each hypothesized 
model (see Figure 1) and the data. Analyses were conducted using Mplus statistical software. Version 
7.0 (Mutlien & Muthen, 2016). An advantage of CFA is that the analyses produce a number of fit 
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statistics, making it possible to evaluate how well the hypothesized model fits the observed data. We 
examined fit indices including the chi-squared statistic, the Comparative Fit Index (CFI) and the 
Root Mean Square Error of Approximation (RMSEA). While small misspecifications in a model 
often lead to inflation of the chi-squared statistic even when the model is correctly specified (Curran, 
West, & Finch, 1996), the CFI accounts for sample size, with a value above 0.90 indicating a good fit 
between the model and the data (Hu & Bender, 1999). The RMSEA indicates how well the model 
would fit the population covariance matrix given unknown but optimally chosen parameters; the 
suggested upper-limit cutoff is 0.06 (Hu & Bender, 1999). 

The robust weighted least squares (WLSMV) estimator was used to estimate the models 
described here. The WLSMV estimator is most appropriate in this research given that almost all 
observed variables are categorical in nature. This estimator uses the probit link and provides 
standard errors and a chi-squared test statistic that are robust to non-normality. 

Results 

Dimensional Structure 

The initial correlated traits CFA model (Model A [original]; see Figure 1) included 10 latent 
factors, as described above, each as measured by its originally hypothesized indicators, per Table 1. 
Results (Table 1) suggested an adequate fit between this model and the data ()F(734)=2181.50; 
CFI=.92; RMSEA=.06). However, a review of factor loadings revealed three (CB PACE4, PERS 
INSTR5, and FLEX2) that fell at or below the recommended .40 threshold (Stevens, 2009), 
suggesting that these indicators should be removed from the model. We also reviewed model 
modification indices, which indicate how the addition or removal of specific parameters would 
improve model fit. Because modification indices are purely data-based and not informed by theory, 
over-reliance on modification indices can quickly become an exploratory journey, increasing the risk 
of Type I error. Thus, no more than a few modifications should be made to an initial confirmatory 
model, provided that a clear theoretical justification exists (McDonald & Ho, 2002). 

The modification indices for the original model included many potential changes. However, 
we identified just one change as consistent with the literature on competency-based education. The 
change involved modeling an original indicator (CB PACE1, “I know what I need to do to show my 
teachers that I am making progress on each competency”) of the competency-based pacing latent factor 
as an indicator of the student demonstration of ownership over learning factor instead. This change is 
consistent with the suggestion that students must be prepared to guide their own learning under 
competency-based education (Le et al., 2014; Lewis et al., 2014). In total, four changes were made to 
the original model: three indicators were dropped from the model, and CB PACE1 became a fourth 
indicator of student demonstration of ownership over learning (OWN4; see Table 1). 

This revision to the original model (Table 1, Model A [revised]), which reflects the changes 
described above (changes in bold font in Table 1), fit the data well (% 2 (618)=1429.61; CFI=.95; 
RMSEA=.05). All factor loadings in the revised model were statistically significant at p< .001. 
Correlations among the 10 latent factors are provided in Table 2. Our next step was to estimate a 
second-order factor model, a model in which the first-order latent factors in turn become indicators 
of a second-order factor. The initial second-order factor model (Model B; see Figure 1) is most 
consistent with current descriptions of competency-based education as comprising four essential 
elements. This model included three second-order factors, representing three of the four key 
elements of competency-based education: mastery-based progression , as indicated by the first-order 
factors CP PACE and CB FEED; personalisation, as indicated by the first-order factors PERS APP 
and PERS INSTR; and skills and dispositions , as indicated by the first-order factors RSPCT, PRSVR, 
RSPNS, OWN, and COLL. We did not model the fourth key element of competency-based 
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education, flexible assessment , as a second-order factor given that this element was reflected through 
just one dimension (FLEX). 


Table 1 


Standardised Loadings on Latent Factors 'Reflecting the Constructs of Interest and Model Fit Statistics 


Model A 
(original) 

P 

Model A 
(revised) 

P 

Model B 

P 

Model C 

P 

CB PACE by 








CB PACE1 

.57 

OWN4 

.57 


.57 


.57 

CB PACE2 

.67 


.66 


.66 


.66 

CB PACE3 

.63 


.62 


.62 


.62 

CB PACE4 

-.22 

omit 

— 


— 


— 

CB PACE5 

.72 


.71 


.71 


.71 

CB FEED by 








CB FEED1 

.81 


.81 


.82 


.82 

CB FEED2 

.79 


.79 


.78 


.79 

CB FEED 3 

.73 


.72 


.73 


.72 

PERS APP by 








PERS APP1 

.56 


.56 


.55 


.54 

PERS APP2 

.97 


.95 


.96 


.95 

PERS APP3 

.76 


.62 


.62 


.62 

PERS APP4 

.72 


.59 


.58 


.58 

PERS APP5 

.94 


.98 


.99 


.99 

PERS INSTR by 








PERS INSTR1 

.63 


.63 


.63 


.63 

PERS INSTR2 

.69 


.69 


.69 


.69 

PERS INSTR3 

.74 


.74 


.74 


.74 

PERS INSTR4 

.71 


.70 


.70 


.70 

PERS INSTR5 

.40 

omit 

— 


— 


— 

PERS INSTR6 

.60 


.59 


.59 


.59 

FLEX by 








FLEX1 

.63 


.61 


.61 


.61 

FT EX?. 

-.29 

omit 

— 


— 


— 

FLEX3 

.63 


.62 


.62 


.62 

FLEX4 

.49 


.42 


.42 


.42 

FLEX5 

.55 


.48 


.47 


.47 

FLEX6 

.61 


.59 


.60 


.60 

FLEX7 

.64 


.63 


.63 


.63 

RSPCT by 








RSPCT1 

.82 


.83 


.82 


.82 

RSPCT2 

.92 


.92 


.93 


.93 

RSPCT3 

.89 


.89 


.89 


.89 



Investigating Student Exposure to Competency-Based Education 


13 


Table 1 (Cont’d.) 

Standardised Leadings on Eatent Factors Reflecting the Constructs of Interest and Model Fit Statistics 


Model A 


Model A 





(original) 

P 

(revised) (3 

Model B 

P 

Model C 

P 

PRSVR by 

PRSVR1 

.88 

.89 


.89 


.89 

PRSVR2 

.87 

.87 


.87 


.87 

PRSVR3 

.82 

.82 


.82 


.82 

RSPNS by 

RSPNS1 

.85 

.85 


.84 


.84 

RSPNS2 

.86 

.86 


.86 


.85 

RSPNS3 

.49 

.49 


.49 


.49 

OWN by 

OWN1 

.76 

.75 


.75 


.75 

OWN2 

.67 

.66 


.65 


.66 

OWN 3 

.76 

.75 


.75 


.75 

COLL by 

COLLI 

.92 

.92 


.92 


.92 

COUP. 

.79 

.79 


.79 


.79 

COLL3 

.80 

.80 


.80 


.80 

Second-order factors 




MB PROG by 


CB EDUC by 





CB PACE 

.88 

CB PACE 

.82 




CB FEED 

.71 

CB FEED 

.67 






PERS APP 

.53 




PERSON by 


PERS INSTR 

.92 




PERS APP 

.55 

FLEX 

.76 




PERS INSTR 

.97 

RSPCT 

.92 






PRSVR 

.81 




SKILLS by 


RSPNS 

.91 




RSPCT 

.93 

OWN 

.73 




PRSVR 

.83 

COLL 

.93 




RSPNS 

.92 






OWN 

.75 






COLL 

.94 



Model fit 

Xfdf 2181.5(734) 

1429.61(618) 

1668.91(648) 

1770.28(653) 

RMSEA 

.060 

.049 


.054 


.056 

(90% Cl) (.057- 

-.063) 

(.046-.052) 

(.051- 

-.057) 

(.051- 

-.057) 

CFI 

.92 

.95 


.94 


.94 

X-diff 



227.34(30), p=.00 

290.21(35), />=. 00 


Note: Source is the Competency-Based Learning Survey for Students, n—599. Bolded font indicates a modification made to 
Model A (original), resulting in Model A (revised). 
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Model B also fit the data well (% 2 (648)=1668.91; CFI=.94; RMSEA=.05). Given that Model 
B is nested within Model A, we used a chi-squared difference test to determine whether Model B fit 
the data as well as Model A. The result was significant (y 2dlt£= 227.34, p— .00; see Table 1), suggesting 
that although both models fit the data well. Model A is preferred over Model B. 

We also estimated an alternative second-order factor model (Model C; see Figure 1) in which 
all 10 first-order latent factors were modeled as indicators of one underlying second-order factor, 
which we might think of as representing competency-based education. Again, this model fit the data well 
(y 2 (653)=1770.28; CFI=.94; RMSEA=.06). Because Model C is also nested within Model A, we 
again used a chi-squared difference test to determine whether Model C fit the data as well as Model 
A. Elere too, the result was significant ()F diff =290.21,y>=.00; see Table 1), indicating that Model A is 
also preferred over Model C. 

Table 2 


Latent ¥ actor Correlations and PReliabilities for Model A (Kevised) 



1 

2 

3 

4 

5 

6 

7 

8 

9 

10 


9=75 

e=. 75 

p=.83 

p=.83 

p=.85 

g=75 

9=75 

9=74 

9=81 

9=75 

1 CB 

PACE 

2 CB 

FEED 

.63 










3PERS 

APP 

4PERS 

.56 

.31 









INSTR 

.84 

.63 

.53 








5 FLEX 

.74 

.50 

.63 

.80 







6 RSPCT 

.45 

.47 

.20 

.58 

.36 






7 PRSVR 

.70 

.54 

.43 

.85 

.66 

.80 





8 RSPNS 

.69 

.63 

.48 

.84 

.70 

.64 

.81 




9 OWN 

.79 

.63 

.34 

.73 

.54 

.59 

.75 

.79 



10 COLL 

.64 

.60 

.43 

.81 

.68 

.75 

.85 

.88 

.69 



Note: Source is the Competemy-Based Naming Survey for Students, n—599. All correlations significant at p<.001. 


In sum, the CFA results suggest that the correlated traits model (Model A), which included 
10 latent factors reflecting different dimensions underlying each of the four elements described as 
essential to competency-based education, fit the data well. Further, all 10 latent factors were 
positively and significantly correlated. The second-order factor models (Models B and C), both of 
which are more consistent with a conceptualization of competency-based education as a bounded, 
distinct model for secondary schooling as opposed to a more arbitrary mix of various dimensions of 
instructional practice, also fit the data well. However, neither second-order factor model fit the data 
as well as the correlated traits model (Model A). We return to the implications of these results in the 
discussion. 

Construct Reliability 

To assess the internal consistency of each of the latent factors, we used the reliability 
coefficient (rho [ 9 ]) developed by Raykov (2001) and recommended (Brown, 2006; Raykov, 2001) 
for use in CFA analyses, particularly when factor indicators are categorical. Constmct reliabilities 
above .70 are considered to reflect high internal consistency (Kline, 2000). All constmct reliabilities 
in the revised Model A ranged from .74—.85 (see Table 2). 
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Convergent and Divergent Construct Validity 

Within a construct validation framework, convergent validity reflects the notion that 
measures we would expect to be related are, in fact, related. In contrast, divergent validity reflects 
the idea that measures we would not expect to be related share only a weak or nonexistent 
relationship. 

Perhaps the most robust evidence of convergent validity is obtained by examining how 
strongly the measures under study are related to previously established measures of the constmcts of 
interest. Given that there are no existing reliable and valid measures of student exposure to 
competency-based education, we were not able to examine this kind of evidence. Although we were 
not able to examine correlations among measures of the same constmcts, we did consider the 
direction and magnitude of the correlations among the latent factors we modeled as another way to 
explore convergent and divergent validity. Almost all correlations among the 10 latent factors were 
positive, moderate or strong, and statistically significant (see Table 2). 

To explore divergent validity, we examined the correlations between each of the 10 latent 
factors and two additional survey items (see Table 3): “Students in my courses all work on the same 
assignment at the same time” and “My teachers spend most of class time giving a lecture or 
presentation to the whole class.” These items were intended to capture the extent to which students 
reported exposure to practices associated with what is often described as a traditional approach to 
instruction, or one that is teacher-directed and assumes that students will all move through the same 
content at about the same pace (Wolfe, Steinberg, & Hoffman, 2013). We anticipated that as student 
exposure to the various dimensions of competency-based education increased, their exposure to 
more traditional instmctional practices would decrease. Generally consistent with this expectation, 
all but one of the latent factors shared a negative or nonexistent relationship with both of these 
survey items. In one exception, the latent factor RSPCT shared a weak positive relationship with the 
item “Students in my courses all work on the same assignment at the same time.” 

Table 3 


Correlations Between Fatent Factors from Model A (Revised) and Two Survey Items Representing a Traditional 
Approach to Instruction _ 



CB 

PACE 

CB 

FEED 

PERS 

APP 

PERS 

INSTR 

FLEX 

RSPCT 

PRSVR 

RSPNS 

OWN 

COLL 

Item 











1 

jg*** 

-.01 


-.05 

-.14*** 

.10* 

-.09* 

12** 

.04 

-.06 

Item 











2 

-.15*** 

_ H** 

-.06 

1j*** 


-.06 

19*** 

-.08 

-.06 

-.10* 


Note : Source is the Con/petemy-Based Eearning Survey for Students, n=599. Item 1 wording is “Students in my courses all work 
on the same assignment at the same time” and Item 2 wording is “My teachers spend most of class time giving a lecture 
or presentation to the whole class” (response options for both are 0=Never, l=Seldom, 2=Sometimes, 3=Often, 

4=Always). p<.05*, p<.01**, p<. 001*** 

The positive and moderate to strong correlations among most of the latent factors are 
potentially indicative of convergent validity, or the idea that measures that should be related are 
actually related. The negative correlations these factors shared with items representing a more 
traditional approach to teaching and learning are suggestive of divergent validity, or the idea that 
measures that should not be related are indeed not related. However, we note that these results are 
exploratory and somewhat difficult to interpret without additional research. 
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Construct Generalizability 

Another aspect of construct validity is generalizability (Messick, 1995). Multigroup CFA can 
be used to test whether the constructs of interest are generalizable across populations (Sass, 2011) 
or, in other words, whether the measures under study are invariant across populations. Although this 
step is often overlooked, it is important to establish measurement invariance (i.e., the same 
constructs can be measured in the same way across populations) before attempting to draw 
comparisons across different populations—for example, different schools—on the constructs of 
interest (Sass, 2011). This research used data from two different high schools, one that was just 
beginning to implement competency-based learning and one that had been competency-based for 
several years, providing an opportunity to test whether our measures were invariant across these two 
groups using multigroup CFA. Muthen and Asparouhov (2002) assert that in order to make 
meaningful comparisons of factor distributions across groups, a majority of the variables serving as 
latent factor indicators should have both threshold and loading invariance. The criteria suggested by 
Muthen and Asparouhov were satisfied (see Appendix for details), indicating that the measures in 
Model A (revised) were invariant across the two schools. 

Discussion 

Interest in competency-based and other student-centered reforms at the secondary level 
continues to grow, but research lags behind (Freeland, 2014; Flaynes et al., 2016). As one 
consequence, changes in educational policy and practice associated with the shift to competency- 
based education often lack empirical evidence. One obstacle for both researchers and practitioners 
interested in better understanding student experiences and outcomes under competency-based 
education has been the lack of tools that could be used to measure and study variation in 
implementation. Through the testing of a survey that captures student-reported exposure to 
hypothesized essential elements of competency-based education, this study begins to address the 
need for measurement instmments that will allow researchers to investigate how this ambitious, if 
somewhat nebulous, reform is actually being implemented, and with what consequence for student 
outcomes. 

Overall, results from the confirmatory factor analyses indicated that the correlated traits 
model (Model A, revised) provided the best fit with the data. In this model, various dimensions of 
the four hypothesized elements of competency-based education were represented by 10 latent 
factors, all of which demonstrated high internal consistency. The model allowed for correlations 
among these 10 latent factors, but we did not group them into one or more higher-order factors. We 
also examined two second-order factor models. The first (Model B) modeled the first-order latent 
factors as indicators of second-order factors representing elements of competency-based education; 
the other (Model C) modeled all 10 first-order latent factors as indicators of a single competency- 
based education second-order factor. The second-order models reflected two ways of representing 
competency-based education as a bounded, distinct model for secondary schooling, as opposed to a 
more arbitrary mix of various dimensions of instmctional practice (Model A). Although both 
second-order factors models fit the data well, results from chi-squared difference tests indicated that 
neither model provided a better fit to the data than Model A. Conceptually, these results indicate 
that the survey reliably captures distinct dimensions of instructional practice associated with 
competency-based education. But the results also indicate that these dimensions may not cluster 
together in a manner consistent with the description of competency-based education as a distinct 
model that can be distinguished from other approaches to secondary education. We return to this 
point below in discussing directions for both future research and policy. 
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With respect to construct validity, we worked closely with a group of six advisors that 
included practitioners and researchers with expertise in competency-based education to establish 
both face and content validity of the survey instrument. We also considered the extent to which our 
results provided preliminary evidence supporting convergent and divergent validity of the constructs 
operationalized in this research. Without existing reliable and valid measures of student exposure to 
elements of competency-based education, we were not able to examine how our measures correlated 
with other established measures, which would have provided the most robust evidence about 
convergent validity. We did, however, examine the direction and magnitude of correlations among 
the latent factors we modeled, and almost all of the correlations among the 10 latent factors were 
positive, moderate or strong, and statistically significant. Nonetheless, we emphasize that these 
results are exploratory. It is difficult to draw firm conclusions about constmct validity, particularly 
given the limited empirical basis upon which to base expectations about the distinctive elements of 
competency-based education—including the component dimensions of those elements—and the 
relationships among them. As we discuss below, future research will be necessary to more 
comprehensively evaluate convergent and divergent validity and other aspects of constmct validity. 

For both practitioners and researchers, results from this research suggest that the survey 
offers a way to reliably measure and study variation in the implementation of practices commonly 
associated with competency-based education. Importantly, the survey provides a way to capture 
implementation from the student perspective. In the remaining sections, we outline implications for 
practice, for future research that can provide needed empirical evidence about a school reform 
model that is increasingly making its way into state and district policy, and, finally, we discuss 
implications for policy itself. 

Implications for Practice 

Given that student reports about their classroom experiences appear to be a particularly 
reliable source of information about instructional practice (Kane & Staiger, 2012), student-reported 
exposure to competency-based education may be especially useful as schools seek to understand and 
refine instruction. Results from student perception surveys such as this one can also provide 
feedback for the improvement of learning and instruction (Ferguson, 2010), and can be used to 
inform data-based decision making (Lai & Schildkamp, 2013). For example, where students report 
low exposure to particular dimensions of instmctional practice associated with competency-based 
education, this could be useful feedback for teachers and schools as they work toward more 
consistent implementation of related practices. It may also be valuable for practitioners and 
policymakers to consider differences among particular student subgroups in their reported exposure 
to dimensions of competency-based education. Although competency-based education has been 
described as uniquely capable of addressing the academic and social-emotional needs of students 
who struggle under traditional systems of schooling, whether this is the case remains an open 
question (Lewis et al., 2014). 

Further, either collectively or in collaboration with researchers, practitioners might also 
triangulate other sources of evidence about instructional practice, for example from classroom 
observations or staff surveys, with student-reported experiences. This type of analysis would provide 
better information about adherence, or the extent to which hypothesized elements of competency- 
based education are implemented as prescribed in the literature at both the classroom and school 
levels. Such an analysis would also provide information about the extent to which students’ 
perceptions of instructional practices under competency-based education parallel the perceptions of 
their teachers. Instances of substantial misalignment between student and teacher perceptions about 
what is happening in the classroom would provide useful opportunities for school staff to reflect 
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together on the sources of this disconnect. Potential sources might include, for example, student 
and/or teacher over- or underreporting of competency-based instructional practices or nuances in 
the wording of student survey items. 

Implications for Research 

With respect to future research, a next step should be to administer this survey across more 
schools both to assess reliability within schools and to measure variation in how students experience 
competency-based education among schools and districts. As noted above, subsequent survey 
administration could be carried out in conjunction with qualitative research, including both 
observations and interviews in schools, to document the existence of certain practices and compare 
these data with student feedback. Combining student survey administration with qualitative data 
collection could also prove valuable for evaluating how much variation in student responses derives 
from measurement issues as opposed to real variation in their classroom and school experiences. 

There is also a need for continued research investigating the program differentiation aspect 
of implementation fidelity, or the degree to which the essential elements of competency-based 
education are distinguishable from each other and from other programs. We considered program 
differentiation in this research by comparing how well three potential ways of modeling 
competency-based education implementation fit the observed data. The initial second-order model 
conceptualized the individual dimensions as reflecting underlying elements of competency-based 
education, and the other second-order model conceptualized these dimensions as reflecting a 
singular underlying “competency-based education” constmct. While both second-order models fit 
the data well, neither fit the data as well as the initial model in which the dimensions of each element 
were modeled as correlated but separate. On the one hand, it is possible that we have not fully 
captured the elements of competency-based education through the dimensions we measure and, 
were we able to do so, we would find that the separate dimensions (i.e., first-order factors) actually 
are better modeled as representing one or more of the hypothesized elements (i.e., second-order 
factors). On the other hand, the results may instead suggest that these dimensions do not cluster 
together in a manner that supports the conceptualization of competency-based education as a 
distinct model with four constituent elements. In other words, competency-based education may 
actually unfold as a more arbitrary mix of instructional practices that, at the level of implementation, 
are combined in ways that can’t necessarily be distinguished from other models for secondary 
education—a possibility consistent with research indicating that whole-school models often lack the 
specificity required to achieve, and to measurably detect, high implementation fidelity (Desimone, 
2002; O’Donnell, 2008). 

Based on these findings, future research on program differentiation might use this survey 
with students from a larger sample of schools to study whether elements described as essential to 
competency-based education actually constitute a coherent and distinct model for secondary 
education. This could include studies that administer the survey both with students in schools that 
have adopted a competency-based model and with students in schools that have not adopted this 
approach. Survey administration with both types of students would provide evidence about the 
extent to which the survey can be used not only to distinguish among schools demonstrating 
different levels of implementation fidelity, but also to distinguish between schools that have and 
have not adopted a competency-based approach. Student survey results could also be compared 
with information collected through other modes (e.g., teacher surveys, classroom observations) in 
order to examine whether different sources of information about program differentiation, and about 
other aspects of implementation fidelity, provide similar or different results. 
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Future research should also continue to explore the validity of constructs reflecting student 
exposure to hypothesized elements of competency-based education, including the validity of 
constructs that attempt to combine these elements into one overall measure of competency-based 
education implementation. In the absence of previously established valid and reliable measures of 
student exposure to competency-based education, our efforts to assess construct validity were 
limited to examining the direction and magnitude of correlations among the first-order latent factors 
as a way to consider convergent validity. Almost all correlations among the 10 latent factors, which 
reflect different dimensions of the hypothesized competency-based education elements, were 
positive, moderate or strong, and statistically significant. Yet, we emphasize that these results are 
exploratory and difficult to interpret without additional research. A correlation between two 
measures provides a somewhat ambiguous signal about convergent and divergent validity given the 
potential influences of both trait and method variance (Furr & Bacharach, 2013). A strong 
correlation may indicate that two measures share trait variance, meaning that the constmcts that they 
are intended to measure share something in common. Flowever, a strong correlation could also 
indicate that the two measures share method variance, indicating that they are correlated primarily 
because they use the same method of measurement. Future research using multi-trait, multi-method 
techniques (Widaman, 1985) could investigate whether factors defined by different measures (e.g., 
online survey, paper/pencil survey) of the same trait (e.g., dimensions of competency-based 
education) support the validity of the measures reported here, and also whether factors defined by 
different traits but measured with the same method argue for method effects. 

Ultimately, constmct validation involves a perpetual process of refining both theory and 
measurement (Westen & Rosenthal, 2003). A more comprehensive analysis of construct validity 
with respect to competency-based education will require continued research, including iterative 
refinement of both theory and measurement. This should include, for example, collecting 
longitudinal data to facilitate the investigation of predictive validity, which will also require specificity 
about the outcomes we should expect to be especially impacted by exposure to competency-based 
education. At the secondary level, competency-based education is typically proposed as a reform to 
improve students’ college and career readiness—but what does this mean, specifically? For example, 
is competency-based education intended to improve academic achievement as measured by 
standardized assessments, performance on college entrance exams, or other measures of college and 
career readiness, including social and emotional outcomes such as engagement and intrinsic 
motivation? Or is it intended to impact all of the above? 

Finally, this survey, which reliably captures student exposure to various dimensions of 
instructional practice associated with competency-based education, might be used to more carefully 
investigate the relationship between implementation and student outcomes. A handful of studies in 
recent years have attempted to estimate the impact of various student-centered approaches, 
including competency-based, personalized, student-centered, and “deeper” learning approaches 
(Flaynes et al., 2016; Steele et al., 2014; Zeiser, Taylor, Rickies, Caret, & Segeritz, 2014). Yet these 
studies have yielded limited knowledge about the effects of competency-based education due to 
considerable variation in its implementation. For example, in one recent study (Flaynes et al., 2016), 
designed to compare outcomes among students in schools identified a priori as competency-based or 
non-competency-based, the researchers found this to be a problematic distinction. Instmctional 
practices commonly associated with competency-based education in the literature varied 
substantially within competency-based schools; moreover, these same practices were also evident, 
and similarly variable, in schools identified as non-competency-based. The researchers concluded 
that competency-based education cannot be characterized as a present/non-present phenomenon. 
Rather than treating the implementation of competency-based education as a dichotomous 
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designation, future research might use the survey described in this study to investigate how the 
nature and extent of student exposure to competency-based education impacts student outcomes. 

Implications for Policy 

Although policy prescriptions are best informed by a cumulative body of evidence, rather 
than by any single study (McDonnell, 2000), the results from this research, while preliminary, 
suggest several directions for policy. 

First, we see a need for policy at national, state, and local levels to establish greater precision 
in the language used to define competency-based education and its key elements. As Hess (2016) 
notes, the school reform landscape is littered with words that have been stripped of meaning, which 
is true, to varying degrees, about many of the terms used to define elements of competency-based 
education, and to define student-centered reforms more generally. Ultimately, the lack of clarity in 
what such placeholders mean avoids complicated questions (Hess, 2016), including questions about 
how to coherently define and successfully implement changes in policy and practice, and about how 
to measure and evaluate these changes and their implications for students. As noted earlier, 
imprecision in the language used to describe and define competency-based education and related 
approaches is accompanied by wide variability within and across schools and districts in how 
competency-based policies and practices are conceived and implemented. Thus, it is perhaps not 
surprising that our results did not necessarily support the conceptualization of competency-based 
education as a distinct model with four constituent elements, at least as these elements have 
currently been defined. 

Second, the results, which raise numerous questions for future research, highlight the fact 
that greater empirical evidence about the nature and effectiveness of competency-based education is 
needed to inform both policy and advocacy at both the national and state levels. Most U.S. states 
have now authorized at least some flexibility for local education agencies to link the completion of 
academic credit to demonstration of proficiency rather than simply to seat time (Carnegie 
Foundation for the Advancement of Teaching, 2014). Yet, there remains little available empirical 
evidence about the effect of competency-based education on student outcomes (Freeland, 2014; 
Haynes et al, 2016; Le et al., 2014). Moving forward, policies at all levels of the education system 
related to competency-based education would benefit from a stronger grounding in evidence from a 
variety of sources and using a range of methods. Related, proponents of competency-based and 
other student-centered educational reforms can best support sound policy by engaging in evidence- 
driven advocacy—even, or perhaps especially, when the evidence challenges prevailing assumptions 
about how well these reforms work (McCarthy, 2016). 

Third, the results draw attention to the fact that input from students is an important source 
of information as state and especially district policymakers seek to understand how competency- 
based education is being implemented in the classroom, and student voices can provide guidance 
about where changes in policy might be necessary. Recent research about the use of student 
perception surveys to gather data about students’ classroom experiences has demonstrated that 
student surveys may provide particularly consistent feedback about what is happening in the 
classroom (Balch, 2012; Ferguson, 2012; Kane & Staiger, 2012), offering an important source of 
information about implementation. While much of this previous research has been situated in the 
literature on teacher evaluation, we believe the survey studied in this research is best used not as an 
accountability tool, but rather as a source of information about where changes in policy may be 
needed to clarify the defining features of competency-based education and to support 
implementation. For example, during our discussions with each of the participating districts about 
their survey results, leaders and teachers identified specific patterns that, from their perspective, 
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indicated a need for both changes in policy and changes in how existing policies were communicated 
to teachers and students. 

Finally, we remind the reader that policy changes, while necessary, are almost certainly 
insufficient to achieve successful implementation of competency-based education. As Spady and 
Mitchell (1977) wisely advised several decades ago, doing so will also require attending to the social 
order of the school, for example by developing a sense of ownership for the proposed 
implementation among students, staff, and parents, by providing rewards to teachers and students 
for risking new approaches to their respective roles, and by encouraging greater authenticity in the 
relationships and goals of students and teachers. 

Conclusion 

Recent evidence from a nationally representative sample of high school graduates indicates 
that almost half of U.S. high school students complete neither a college- or career-ready course of 
study (Bromberg & Theokas, 2016). Policymakers, employers, and the public voice concerns that 
America’s youth are graduating without the skills and knowledge needed for postsecondary success. 
In response, many states and districts are turning to competency-based education policies and other 
so-called student-centered reforms as a strategy to ensure student readiness for college-level work 
and the workforce. 

As competency-based and other student-centered reforms increasingly make their way into 
policy, empirical research lags behind. Research must keep pace with changes in policy and practice 
by providing evidence about both the implementation and the effects of competency-based reforms. 
This should include research that builds upon the current study to investigate the link between 
student-reported exposure to competency-based education and student outcomes. 
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Appendix A 


Table Al. Sample Descriptive Statistics on Survey Items, Organised by Original Constructs (Constructs Measured 
by Patent Eactors) 


Item 

Construct and Item description 

M(SD) 

CB PACE1 

Mastery-based progression (CB PACE), as indicated by how often 
each of the following occurs in the student’s current courses 3 : 

I know what I need to do to show my teachers that I am making 

2.64(0.81) 

CB PACE2 

progress on each competency 

I must show my teachers that I have mastered each competency 

1.86(1.26) 

CB PACE3 

before I can move on to the next one 

I am able to move on to the next competency when I am ready, 

1.41(1.24) 

CB PACE4 

even if other students in the course are not ready 

Students in my courses work on the same competency at the same 

3.12(0.93) 

CB PACE5 

time 

I understand how the competencies in my courses will help me in 

2.08(1.05) 

CB FEED1 

the future 

Competency-specific feedback (CB FEED), as indicated by how 
often each of the following occurs in the student’s current 
courses 3 : 

My teachers share examples of excellent work on each 

2.08(0.91) 

CB FEED2 

competency 

My teachers let me know how my work will be assessed or graded 

2.54(0.95) 

CB FEED 3 

for each competency 

My teachers give me a mbric so that I know how I am progressing 

2.70(0.95) 

PERS APP1 

on each competency 

Personalization: Credit (PERS APP), as indicated by the share of 
the student’s current courses in which each of the following is 
true b : 

I am able to complete some or all course requirements online 

0.36(1.02) 

PERS APP2 

If I complete a project that wasn't assigned at school but is related 

0.46(0.65) 

PERS APP3 

to a course I am taking, I can earn credit for the project in that 
course 

I can earn credit for taking courses at another high school 

0.86(0.70) 

PERS APP4 

I can earn credit for taking courses at a college 

1.00(0.68) 

PERS APP5 

I can earn credit for completing an internship or job shadowing 

0.73(0.73) 
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Table A1 (Cont’d.) 

Sample Descriptive Statistics on Survey Items, Organised by Original Constructs (<Constructs Measured by Latent 
Factors) 


Item 

Construct/Item description 

M(SD) 


Personalization: Individual (PERS INSTR), as indicated by how 
often each of the following occurs in the student’s current 
courses 3 : 


PERS INSTR1 

My teachers work with students in small groups or individually 

1.86(0.90) 

PERS INSTR2 

My teachers notice if I need extra help 

2.00(1.03) 

PERS INSTR3 

My teachers teach the material in several different ways in order to 
help students learn 

1.80(1.04) 

PERS INSTR4 

My teachers and I discussed how I am doing on each competency 

0.97(0.90) 

PERS INSTR5 

My teachers gave me written feedback on my work 

1.90(0.88) 

PERS INSTR6 

I have had opportunities to choose how to show my teachers 
what I have learned 

1.37(1.02) 


Flexible assessment (FLEX), as indicated by the number of times 
each of the following occurred in the students current courses 3 : 


FLEX1 

I have created drawings or models to show what I have learned 

0.90(0.85) 

FLEX2 

I have taken tests or quizzes to show what I have learned 

2.63(0.68) 

FLEX3 

I have given a performance to show what I have learned 

0.83(0.90) 

FLEX4 

I have given a presentation to show what I have learned 

1.38(0.86) 

FLEX5 

I have completed a project at school to show what I have learned 

1.52(0.87) 

FLEX6 

If I do poorly on an assessment on the first try, I can try again 

2.07(1.03) 

FLEX7 

To show that I have mastered a course competency, I must 
demonstrate my learning in more than one way 

1.86(1.07) 


Skills and dispositions: Respect (RSPCT), as indicated by how 
often each of the following occurs in the student’s current 
courses 3 : 


RSPCT1 

Teachers encourage students to respect the feelings of others 

2.62(1.13) 

RSPCT2 

Teachers show students how to treat each other with respect 

2.38(1.14) 

RSPCT3 

Teachers explain to students how they can disagree with each 
other in a respectful way 

2.22(1.12) 
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Table A1 (Cont’d.) 

Sample Descriptive Statistics on Survey Items, Organised by Original Constructs (<Constructs Measured by Fatent 
Factors) 


Item 

Construct/Item description 

M(SD) 


Skills and dispositions: Perseverance (PRSVR), as indicated by 
how often each of the following occurs in the student’s current 
courses 3 : 


PRSVR1 

When I have trouble learning something new, my teachers give 
me advice and strategies that help me to stick with it 

2.10(1.04) 

PRSVR2 

My teachers notice when I take extra time and effort on 
something that is difficult for me 

1.95(1.17) 

PRSVR3 

If I get a low score on an assessment, my teachers help me figure 
out how I can still do well in the class 

2.08(1.17) 


Skills and dispositions: Responsibility (RSPNS), as indicated by 
how often each of the following occurs in the student’s current 
courses 3 : 


RSPNS1 

Teachers show students how to keep track of their progress on 
each of the competencies 

1.84(1.09) 

RSPNS2 

Teachers show students strategies for making sure all assignments 
and assessments are completed on time 

1.90(1.01) 

RSPNS3 

Teachers encourage students to take responsibility for their work 

2.83(0.90) 


Skills and dispositions: Ownership (OWN), as indicated by how 
often each of the following occurs in the student’s current 
courses 3 : 


OWN1 

I know which steps to take during high school in order to prepare 
for what I want to do after I graduate 

1.86(1.03) 

OWN2 

If I need information that I don't have in order to complete an 
assignment, I know where to get it 

1.89(1.02) 

OWN 3 

If it is difficult for me to get an assignment done on my own, I 
know strategies I can use to get the work finished 

1.77(1.09) 


Skills and dispositions: Collaboration (COLL), as indicated by 
how often each of the following occurs in the student’s current 
courses 3 : 


COLLI 

Teachers show students how to help each other learn 

2.23(1.10) 

COT 1.2 

Teachers explain to students how to work together in groups 

2.48(0.96) 

COLL3 

Teachers encourage students to help each other outside class 

2.35(1.04) 


Note: Source is the Competenty-Based Learning Survey for Students, n=599. a Response options included 0=Never, l=Seldom, 
2=Sometimes, 3=Often, 4=Always. b Response options included 0=None, l=Some, 3=A11 or most. c Response options 
included 0=Not at all, 1=1—2 times, 2=3—4 times, 3=5 or more times. 
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Appendix B 

Measurement Invariance Testing Procedures 

Per the recommended procedure for establishing measurement invariance (Muthen & Muthen, 

2016), we began with a model in which item thresholds and factor loadings were freely estimated 
across the two groups, with scale factors fixed at one in both groups and factor means fixed at zero. 
This model fit the data well (% 2 (1236)=1838.40; CFI=.96; RMSEA=.04). We then estimated a model 
in which thresholds and factor loadings were constrained to be equal across the two groups. 
Although this model fit the data well (% 2 (1351)=2042.23; CFI=.96; RMSEA=.04), the chi-square 
difference test value was significant (y 2 dltt (l 15)=280.75, p—. 00), meaning that constraining all of 
these parameters significantly worsened the fit of the model. Modification indices suggested allowing 
the thresholds for two items (CB PACE5 and FLEX6) to be freely estimated in both groups. After 
making this change, this model also fit the data well (% 2 (1339)=1895.69; CFI=.96; RMSEA=.04), and 
the chi-square difference test value was non-significant (}F dlff (103)=109.33,y>=.31). 
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